Serverless Jenkins with Jenkins X

Jenkins server was originally created in 2004 in the form of Hudson. Jenkins has become a household name for so many of us in software development and delivery and is the leader in continuous integration plus continuous delivery tooling. To date there are over 20.5 million Jenkins jobs and close to 200,000 Jenkins servers running. These are amazing numbers!

During that growth there has been significant advances in technology, such as the cloud and containerization, which means some of the responsibilities of Jenkins now have better implementations we should look to leverage. These days most companies have cloud initiatives and we want Jenkins to evolve with the times and take it’s own cloud native journey. Jenkins should continue to grow and provide the automation, reliability and developer experience that is relied upon by so many.

The tremendous success of Jenkins has also come with pain points. Let’s quickly recap on some of the biggest issues we hear about:

• The Jenkins server is a single point of failure (SPOF), in particular git webhook events are missed during any maintenance downtime.

• Jenkins servers frequently run out of disk space and require scripting and/or manual cleanups by people to keep the lights on.

• Plugin version mismatches can cause conflicts during upgrades.

• GitHub rate limiting, caused by multibranch plugin scans.

• Large Java virtual machine (JVM) requiring high memory, even when no builds are running, results in unwanted costs when using usage-based pricing.

What if:

• We could reduce our cloud bills by only running Jenkins to process our pipelines when a build is required.

• An ephemeral pipeline engine could be discarded after a build is finished, which avoids the filesystem filling up and eventually running out of disk space.

• Continuous integration could validate the installation of new Jenkins plugins or plugin version upgrades.

• Provide high availability and a scalable webhook handler to address the SPOF.

• Avoid GitHub API scans reducing risk of being rate limited.

• Provide disaster recovery strategy where all job configuration is stored in Git.

The Jenkins X project was announced earlier this year to provide automated CI + CD for Kubernetes with Preview Environments for Pull Requests and automatic GitOps promotion through your environments (Testing -> Staging -> Production). Jenkins X also extends Kubernetes using CRDs (custom resource definitions ) and orchestrates your Jenkins servers and pipelines.

The Jenkins X and Jenkins projects are now thrilled to announce Serverless Jenkins! Jenkins X can now orchestrate either Serverless Jenkins, a static Jenkins controller or Knative builds per team; so it’s now your open source Jenkins cloud with full Knative build support!

Serverless Jenkins uses successful and innovative open source projects to address the issues identified above with static Jenkins controllers. Kubernetes is now the de facto cloud implementation so let’s focus on the lesser known projects that also make Serverless Jenkins possible: Prow and Knative build.

At the bottom of this blog is a link to an unedited YouTube recording that demonstrates this in action.

Prow comes from the Kubernetes ecosystem and was created by the fine folks at Google when they began to struggle with using Jenkins across the Kubernetes GitHub repos. Kubernetes is one of the largest and most successful open source projects on GitHub, and Prow is used by all their 140 repos as well as for Istio and Jetstack . It is an event-based solution which is made up from a number of microservices each having distinct responsibilities — providing a loosely coupled architecture ideal for a cloud native architecture. It gives powerful controls over our merges to controller (both before, and after the pull request build runs) and uses ChatOps to interact with a build system.

Prow includes a scalable and highly available webhook event handler that writes ProwJob CRDs to Kubernetes based on Git events, so that other microservices (Kubernetes controllers that ‘watch’ for these ProwJob events) can react and perform operations such as running continuous integration or delivery pipelines. These Git events can be triggered by new PRs and issues, comments, merges, pushes etc so we can react to a wide set of triggers.

It also includes the ability to automatically merge pull requests given a certain set of configurable rules based on labels.

For a list of Prow components and description take a look at this repo .

Prow also stores its job configuration in Git. This means in a disaster recovery situation all CI and CD jobs can be restored. To see an example of this the Jenkins X project as always has adopted this first to ensure we dogfood and validate before rolling it out to users. You can see the Jenkins X projects own Prow configuration for every repo we have that requires CI/CD here .

Knative Build is another cloud native solution that extends Kubernetes with CRDs and provides users ways to build applications from source. One of the great features of Knative Build is that you can string together simple steps that run sequentially in the same Kubernetes pod, using different containers where you can share state between each step. This approach uses Kubernetes init containers .

Build Templates are a way to reference the type of Kubernetes pod that’s created to run your builds. They allow you to specify which docker image to execute your build in, what environment variables should exists at build time and what service accounts, secrets and volumes should be mounted. Build templates are Kubernetes CRDs and are automatically upgraded with each Jenkins X release. The Prow configuration that’s generated by Jenkins X when creating or importing applications references a Build Template. An example on the Jenkins X project is the prow config pointing at a Build Template .

Now that we understand the background and context of what we are doing here, now we can look at what Serverless Jenkins is. It’s part of the Cloud Native Jenkins effort to help developers, teams and organizations move to the cloud and ensure that Jenkins is not only relevant for the cloud and allows us to leverage what both the cloud AND Jenkins do best.

Using Jenkins X on Kubernetes will automatically install and configure Prow and Knative for you so that you are ready to go from the install. The jx CLI generates all the configuration needed and updates Git repo webhook endpoints when creating or importing projects.

Now each pull request or merge to controller triggers spinning up an ephemeral Jenkins in Kubernetes using Knative, checkouts the Git revision, configures required credentials and runs the applications build pipeline using its Jenkinsfile . It will then discard the Jenkinsfile runner pod once the build is finished.

Thanks to the Custom War Packager (CWP) the Jenkins X release process builds different flavours of Jenkins servers that contains the necessary build tools. Language detection ensures that the right flavour is used. We also use the Configuration as Code plugin (CasC) to add the necessary Jenkins config at build time. One awesome feature of CWP is it extracts the Jenkins plugins during the build of Serverless Jenkins (and not when Serverless Jenkins starts), so the container and JVM start time for a Jenkins X based Jenkins image is sub 5 seconds — in comparison, it can take several minutes to start a Jenkins server on Kubernetes.

We have a monorepo which is used to automatically build and release these language specific Jenkins images whenever we release Jenkins X.

This also means that because our plugins are defined in yaml and stored in Git we can have CI and CD for our CI and CD tooling. When we want to upgrade a plugin we make a pull request which triggers CI and builds a preview Jenkins image, ensuring there is no plugin conflicts and we can even run mock jobs as automated testing (although we have not done this part yet). And everyone can take the exact same approach and build custom Serverless Jenkins images to use in their pipelines in the same way.

One thing to highlight is that when you switch to Serverless Jenkins there’s no state stored between builds (meaning that the build number for each job would always be 1). In Jenkins X we create a CRD for the PipelineActivity so we can generate the next build number and also store information about the build which allows us to visualize previous builds pipelines after the single shot Jenkins build has completed.

When Prow receives a webhook event it will create a Knative build resource in Kubernetes. Next the Knative build controller that’s watching for builds will create a Kubernetes pod and automatically add a init container that clones the PR or release branch source code. Next, leveraging the Jenkinsfile runner , a single shot Jenkins is started in a separate step which can access the source code cloned by Knative and process the application’s Jenkinsfile.

Today Jenkins X includes Prow out of the box when creating clusters on GKE with terraform via

jx create terraform

Or when using the feature flag on other create cluster or install commands, i.e.

jx create cluster gke --prow

jx install — prow

Q. So if there’s no static Jenkins server running, how do I access the GUI?

A . There is no open source Jenkins GUI for Serverless Jenkins. This is pretty significant so let’s try and explain. Jenkins X has IDE and CLI tooling that makes working with Jenkins X developer friendly. However, the GUI is no longer available. Prow has an open source GUI called Deck which Jenkins X installs out of the box.

Q. Where do I get build logs?

A . better solution will be worked on, but for now Jenkinsfile runner sends builds log to standard out allowing us to leverage the Kubernetes clusters centralized logging solution such as Stackdriver, CloudWatch. We also provide `jx logs -k` (available whilst the build runs) and `jx get build log` (available for a few hours)

Q.Do I need to change my Jenkinsfiles that rely on specific Jenkins multibranch plugin environment variables like `$JOB_NAME`?

A . No we have tried to ensure all MBP related env vars are still added in the same format. If we have missed any the let us know.

The Jenkins X project itself has already migrated from using a static (always on) Jenkins server to Serveless Jenkins. Yes that’s right, we have scaled down our Jenkins server to 0 and moved all our Git repos to Prow and Serverless Jenkins. You can look at any pull request on the JenkinsX site to see it in action. We were using a declarative style Jenkinsfile (which is what we add when you import a new project into Jenkins X) which means migrating to Serverless Jenkins took only a couple of tweaks to the Jenkinsfile:

1. <>Change the agent type to ‘any’ so that the pipeline is executed on the ephemeral one shot Jenkins <>Remove all Jenkinsfile container blocks as right now it’s assumed all steps are executed in the one shot Jenkins pipeline engine. <>For any release branch pipelines that tag (they all should create a Git tag!) then we have to switch from checkout scm to git ‘github/foo.git’ as there’s a problem with reusing the cloned repo from Knative and the Jenkinsfile runner as there seems to be a symlink used when adding the repo to the Jenkins workspace. We hope to fix this.

An example of the above changes can be seen here . To enable prow’s ChatOps /approve comment then you also need a similar OWNERS file to that link that uses approver GitHub ID’s.

• Currently GitHub only, we will be contributing support for multiple Git providers

• Jenkins X uses a fork however this will be switched back in the next few weeks to use the upstream prow repo

• Jenkins X by default creates a declarative pipeline Jenkinsfiles, this has not been tested on scripted and shared library Jenkinsfile pipelines but we are keen for feedback if this works as expected.

• Kubernetes Plugin PodTemplates are not supported yet. We’re not sure if it’s a good idea to or not yet. It means that if you are migrating existing Jenkinsfiles that have multiple different container {…} blocks you need to add each container’s build tools into the single one shot Jenkins created by CWP above.

There’s still work to do, so if you’d like to get involved come and say hi in the Jenkins X Kubernetes slack rooms , help on an issue, or to start with give it a try and let us know how you get on.

Jenkins X is a one stop stop shop for teams orchestrating their static, serverless or Knative build jobs using Prow ChatOps, with automated CI/CD included for Kubernetes workloads and more automation to come.

• Learn more about Jenkins X